Fast Inference and Learning with Sparse Belief Propagation

Even in trees, exact probabilistic inference can be expensive when the cardinality of the variables is large. This is especially troublesome for learning, because many standard estimation techniques, such as EM and conditional maximum likelihood, require calling an inference algorithm many times. In max-product inference, a standard heuristic for controlling this complexity in linear chains is beam search, that is, to ignore variable configurations during inference once their estimated probability becomes sufficiently low. Although quite effective for max-product, during sum-product inference beam search discards probability mass in a way that makes learning unstable. In this paper, we introduce a variational perspective on beam search that uses a approximating mixture of Kronecker delta functions. This motivates a novel variational approximation for arbitrary tree-structured models, which maintains an adaptivelysized sparse belief state—thus extending beam search from max-product to sum-product, and from linear chains to arbitrary trees. We report efficiency improvements for max-product inference over other beam search techniques. Also, unlike heuristic methods for discarding probability mass, our method can be used effectively for conditional maximum likelihood training. On both synthetic and real-world problems, we report four-fold increases in learning speed with no loss in accuracy.